Dynamic shared experience recommendations

ABSTRACT

A communication server facilitates a video call between client devices of a plurality of participants. During the video call, the communication server identifies an in-call activity to recommend to the plurality of participants in the video call based on user information. The in-call activity is selected to be relevant to each of the plurality of participants and jointly recommended to the plurality of participants of the video call. To identify the recommended in-call activity, the communication server may determine common interests among the plurality of participants and select an in-call activity that is associated with the common interests. After the recommended in-call activity is selected, an indication of the recommended in-call activity is provided to the client devices to enable the client devices to display a user interface including the in-call activity to the participants during the video call.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/784,105, filed Feb. 6, 2020, which claims the benefit of U.S.Provisional Application No. 62/943,626 filed Dec. 4, 2019, which areincorporated by reference in its entirety.

BACKGROUND

Network-based video calling is becoming a common way of communicating.Network-based video calling may be provided through social networkingsystems that allow users in the social networking systems to communicateand stay connected through different means such as messaging, postingcontent, and calling.

SUMMARY

This disclosure relates generally to making activity recommendationsduring a video call, and more specifically to making a jointrecommendation to a plurality of participants of the video call.

Video call platforms may provide in-call features to video calls thatenhance audio and visual experiences by allowing users to participate inactivities together despite not being in a same location. In-callactivities can improve user experiences by helping the users feel moreconnected by allowing them to share an experience while on a video callas if they were in the same room sharing the experience. However, with alarge selection of possible in-call features, it can be difficult forusers to select one in-call activity that is relevant to all of theparticipants on the video call.

Embodiments relate to making a joint recommendation for one or morein-call activities to a plurality of participants of a video call. Acommunication server facilitates a video call between client devices ofthe plurality of participants over a network. For each participantassociated with a client device in the video call, the communicationserver retrieves user information. Based on the user information of theparticipants, the communication server determines common interests amongthe participants. The communication server identifies one or morein-call activities associated with the common interests and selects arecommended in-call activity from the identified one or more in-callactivities for the plurality of participants. After selecting therecommended in-call activity, the communication server transmits anindication of the recommended in-call activity to a client device in thevideo call to enable the client device to display a user interfacecontrol for initiating the recommended in-call activity. When thecommunication server receives a selection from the client device toinitiate the recommended in-call activity, the communication serverinitiates the recommended in-call activity on each of the plurality ofclient devices in the video call.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment for a communicationsystem, in accordance with an embodiment.

FIG. 2A is a block diagram of an architecture of a social networkingserver, in accordance with an embodiment.

FIG. 2B is a block diagram of an architecture of a communication server,in accordance with an embodiment.

FIG. 3A is an example user interface of a video call, in accordance withan embodiment.

FIG. 3B is an example user interface of the video call including in-callactivity recommendations, in accordance with an embodiment.

FIG. 4 is a flowchart illustrating a process for selecting an in-callactivity for participants of a video call, in accordance with anembodiment.

The figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesdescribed herein.

DETAILED DESCRIPTION System Architecture

FIG. 1 is a block diagram of a system environment 100 for acommunication system 120. The system environment 100 includes a socialnetworking server 105, one or more client devices 115 (e.g., clientdevices 115A, 115B), a network 110, a communication system 120, and acommunication server 125. In alternative configurations, differentand/or additional components may be included in the system environment100. For example, the system environment 100 may include additionalclient devices 115 or additional communication systems 120.

The social networking server 105 manages information about users of asocial network and activities performed by the users on the socialnetwork. The social networking server 105 allows users of the socialnetwork to connect with others over the network 110 by sharing contentwith other users of the social network and viewing content shared byother users and entities such as businesses or organizations. Users mayaccess the social networking server 105 via client devices 115 orcommunication system 120. Details on the social networking server 105 isdescribed with respect to FIG. 2A.

In an embodiment, the communication system 120 comprises an integratedcomputing device that operates as a standalone network-enabled device.In another embodiment, the communication system 120 comprises acomputing device for coupling to an external media device such as atelevision or other external display and/or audio output system. In thisembodiment, the communication system may couple to the external mediadevice via a wireless interface or wired interface (e.g., an HDMI cable)and may utilize various functions of the external media device such asits display, speakers, and input devices. Here, the communication system120 may be configured to be compatible with a generic external mediadevice that does not have specialized software, firmware, or hardwarespecifically for interacting with the communication system 120.

The client devices 115 are one or more computing devices capable ofreceiving user input as well as transmitting and/or receiving data viathe network 110. In one embodiment, a client device 115 is aconventional computer system, such as a desktop or a laptop computer.Alternatively, a client device 115 may be a device having computerfunctionality, such as a personal digital assistant (PDA), a mobiletelephone, a smartphone, a tablet, an Internet of Things (IoT) device, avideo conferencing device, another instance of the communication system120, or another suitable device. A client device 115 is configured tocommunicate via the network 110. In one embodiment, a client device 115executes an application allowing a user of the client device 115 tointeract with the communication system 120 by enabling voice calls,video calls, data sharing, or other interactions. For example, a clientdevice 115 executes a browser application to enable interactions betweenthe client device 115 and the communication system 105 via the network110. In another embodiment, a client device 115 interacts with thecommunication system 105 through an application running on a nativeoperating system of the client device 115, such as IOS® or ANDROID™.

The communication server 125 facilitates communications of the clientdevices 115 and the communication system 120 over the network 110. Forexample, the communication server 125 may facilitate connections betweenthe communication system 120 and a client device 115 when a voice orvideo call is requested. Additionally, the communication server 125 maycontrol access of the communication system 120 to various externalapplications or services available over the network 110. In anembodiment, the communication server 125 may provide updates to thecommunication system 120 when new versions of software or firmwarebecome available. In other embodiments, various functions describedbelow as being attributed to the communication system 120 can instead beperformed entirely or in part on the social networking server 105 and/orcommunication server 125. For example, in some embodiments, variousprocessing or storage tasks may be offloaded from the communicationsystem 120 and instead performed on the social networking server 105and/or communication server 125.

The network 110 may comprise any combination of local area and/or widearea networks, using wired and/or wireless communication systems. In oneembodiment, the network 110 uses standard communications technologiesand/or protocols. For example, the network 110 includes communicationlinks using technologies such as Ethernet, 802.11 (WiFi), worldwideinteroperability for microwave access (WiMAX), 3G, 4G, 5G, code divisionmultiple access (CDMA), digital subscriber line (DSL), Bluetooth, NearField Communication (NFC), Universal Serial Bus (USB), or anycombination of protocols. In some embodiments, all or some of thecommunication links of the network 110 may be encrypted using anysuitable technique or techniques.

The communication system 120 includes one or more user input devices122, a microphone sub-system 124, a camera sub-system 126, a networkinterface 128, a processor 130, a storage medium 150, a displaysub-system 160, and an audio sub-system 170. In other embodiments, thecommunication system 120 may include additional, fewer, or differentcomponents.

The user input device 122 comprises hardware that enables a user tointeract with the communication system 120. The user input device 122can comprise, for example, a touchscreen interface, a game controller, akeyboard, a mouse, a joystick, a voice command controller, a gesturerecognition controller, a remote control receiver, or other inputdevice. In an embodiment, the user input device 122 may include a remotecontrol device that is physically separate from the user input device122 and interacts with a remote controller receiver (e.g., an infrared(IR) or other wireless receiver) that may integrated with or otherwiseconnected to the communication system 120. In some embodiments, thedisplay sub-system 160 and the user input device 122 are integratedtogether, such as in a touchscreen interface. In other embodiments, userinputs may be received over the network 110 from a client device 115.For example, an application executing on a client device 115 may sendcommands over the network 110 to control the communication system 120based on user interactions with the client device 115. In otherembodiments, the user input device 122 may include a port (e.g., an HDMIport) connected to an external television that enables user inputs to bereceived from the television responsive to user interactions with aninput device of the television. For example, the television may senduser input commands to the communication system 120 via a ConsumerElectronics Control (CEC) protocol based on user inputs received by thetelevision.

The microphone sub-system 124 comprises one or more microphones (orconnections to external microphones) that capture ambient audio signalsby converting sound into electrical signals that can be stored orprocessed by other components of the communication system 120. Thecaptured audio signals may be transmitted to the client devices 115during an audio/video call or in an audio/video message. Additionally,the captured audio signals may be processed to identify voice commandsfor controlling functions of the communication system 120. In anembodiment, the microphone sub-system 124 comprises one or moreintegrated microphones. Alternatively, the microphone sub-system 124 maycomprise an external microphone coupled to the communication system 120via a communication link (e.g., the network 110 or other directcommunication link). The microphone sub-system 124 may comprise a singlemicrophone or an array of microphones. In the case of a microphonearray, the microphone sub-system 124 may process audio signals frommultiple microphones to generate one or more beamformed audio channelseach associated with a particular direction (or range of directions).

The camera sub-system 126 comprises one or more cameras (or connectionsto one or more external cameras) that captures images and/or videosignals. The captured images or video may be sent to the client device115 during a video call or in a multimedia message, or may be stored orprocessed by other components of the communication system 120.Furthermore, in an embodiment, images or video from the camerasub-system 126 may be processed to for face detection, face recognition,gesture recognition, or other information that may be utilized tocontrol functions of the communication system 120. In an embodiment, thecamera sub-system 126 includes one or more wide-angle cameras forcapturing a wide, panoramic, or spherical field of view of a surroundingenvironment. The camera sub-system 126 may include integrated processingto stitch together images from multiple cameras, or to perform imageprocessing functions such as zooming, panning, de-warping, or otherfunctions. In an embodiment, the camera sub-system 126 may includemultiple cameras positioned to capture stereoscopic (e.g.,three-dimensional images) or may include a depth camera to capture depthvalues for pixels in the captured images or video.

The network interface 128 facilitates connection of the communicationsystem 120 to the network 110. For example, the network interface 130may include software and/or hardware that facilitates communication ofvoice, video, and/or other data signals with one or more client devices115 to enable voice and video calls or other operation of variousapplications executing on the communication system 120. The networkinterface 128 may operate according to any conventional wired orwireless communication protocols that enable it to communication overthe network 110.

The display sub-system 160 comprises an electronic device or aninterface to an electronic device for presenting images or videocontent. For example, the display sub-system 160 may comprises an LEDdisplay panel, an LCD display panel, a projector, a virtual realityheadset, an augmented reality headset, another type of display device,or an interface for connecting to any of the above-described displaydevices. In an embodiment, the display sub-system 160 includes a displaythat is integrated with other components of the communication system120. Alternatively, the display sub-system 120 comprises one or moreports (e.g., an HDMI port) that couples the communication system to anexternal display device (e.g., a television).

The audio output sub-system 170 comprises one or more speakers or aninterface for coupling to one or more external speakers that generateambient audio based on received audio signals. In an embodiment, theaudio output sub-system 170 includes one or more speakers integratedwith other components of the communication system 120. Alternatively,the audio output sub-system 170 comprises an interface (e.g., an HDMIinterface or optical interface) for coupling the communication system120 with one or more external speakers (for example, a dedicated speakersystem or television). The audio output sub-system 120 may output audioin multiple channels to generate beamformed audio signals that give thelistener a sense of directionality associated with the audio. Forexample, the audio output sub-system may generate audio output as astereo audio output or a multi-channel audio output such as 2.1, 3.1,5.1, 7.1, or other standard configuration.

In embodiments in which the communication system 120 is coupled to anexternal media device such as a television, the communication system 120may lack an integrated display and/or an integrated speaker, and mayinstead only communicate audio/visual data for outputting via a displayand speaker system of the external media device.

The processor 130 operates in conjunction with the storage medium 150(e.g., a non-transitory computer-readable storage medium) to carry outvarious functions attributed to the communication system 120 describedherein. For example, the storage medium 150 may store one or moremodules or applications (e.g., user interface 152, communication module154, user applications 156) embodied as instructions executable by theprocessor 130. The instructions, when executed by the processor, causethe processor 130 to carry out the functions attributed to the variousmodules or applications described herein. In an embodiment, theprocessor 130 may comprise a single processor or a multi-processorsystem.

In an embodiment, the storage medium 150 comprises a user interfacemodule 152, a communication module 154, and user applications 156. Inalternative embodiments, the storage medium 150 may comprise differentor additional components.

The user interface module 152 comprises visual and/or audio elements andcontrols for enabling user interaction with the communication system120. For example, the user interface module 152 may receive inputs fromthe user input device 122 to enable the user to select various functionsof the communication system 120. In an example embodiment, the userinterface module 152 includes a calling interface to enable thecommunication system 120 to make or receive voice and/or video callsover the network 110. To make a call, the user interface module 152 mayprovide controls to enable a user to select one or more contacts forcalling, to initiate the call, to control various functions during thecall, and to end the call. To receive a call, the user interface module152 may provide controls to enable a user to accept an incoming call, tocontrol various functions during the call, and to end the call. Forvideo calls, the user interface module 152 may include a video callinterface that displays remote video from a client 115 together withvarious control elements such as volume control, an end call control, orvarious controls relating to how the received video is displayed or thereceived audio is outputted.

The user interface module 152 may furthermore enable a user to accessuser applications 156 or to control various settings of thecommunication system 120. In an embodiment, the user interface module152 may enable customization of the user interface according to userpreferences. Here, the user interface module 152 may store differentpreferences for different users of the communication system 120 and mayadjust settings depending on the current user.

The communication module 154 facilitates communications of thecommunication system 120 with clients 115 for voice and/or video calls.For example, the communication module 154 may maintain a directory ofcontacts and facilitate connections to those contacts in response tocommands from the user interface module 152 to initiate a call.Furthermore, the communication module 154 may receive indications ofincoming calls and interact with the user interface module 152 tofacilitate reception of the incoming call. The communication module 154may furthermore process incoming and outgoing voice and/or video signalsduring calls to maintain a robust connection and to facilitate variousin-call functions.

The user applications 156 comprise one or more applications that may beaccessible by a user via the user interface module 152 to facilitatevarious functions of the communication system 120. For example, the userapplications 156 may include a web browser for browsing web pages on theInternet, a picture viewer for viewing images, a media playback systemfor playing video or audio files, an intelligent virtual assistant forperforming various tasks or services in response to user requests, orother applications for performing various functions. In an embodiment,the user applications 156 includes a social networking application thatenables integration of the communication system 120 with a user's socialnetworking account. Here, for example, the communication system 120 mayobtain various information from the user's social networking account tofacilitate a more personalized user experience. Furthermore, thecommunication system 120 can enable the user to directly interact withthe social network by viewing or creating posts, accessing feeds,interacting with friends, etc. Additionally, based on the userpreferences, the social networking application may facilitate retrievalof various alerts or notifications that may be of interest to the userrelating to activity on the social network. In an embodiment, users mayadd or remove applications 156 to customize operation of thecommunication system 120.

FIG. 2A is a block diagram of an architecture of the social networkingserver 105. The social networking server 105 shown in FIG. 2A includes auser profile store 205, a content store 210, an action logger 215, anaction log 220, and an edge store 225. In other embodiments, the socialnetworking server 105 may include additional, fewer, or differentcomponents for various applications. Conventional components such asnetwork interfaces, security functions, load balancers, failoverservers, management and network operations consoles, and the like arenot shown so as to not obscure the details of the system architecture.The social networking server 105 may use a single processor or amulti-processor system in conjunction with a storage medium such as anon-transitory computer-readable storage medium that stores instructionsexecutable by the processor(s) to carry out various functions describedherein.

The social networking server 105 links an online system (e.g., socialnetworking system) via the network 110 to the one or more client devices115, as well as to the one or more communication systems 120. The socialnetworking server 105 serves web pages, as well as other content, suchas JAVA®, FLASH®, XML and so forth. The social networking server 105 mayreceive and route messages between the online system and the clientdevices 115 and the communication system, for example, instant messages,queued messages (e.g., email), text messages, short message service(SMS) messages, or messages sent using any other suitable messagingtechnique. A user may send a request to the social networking server 105to upload information (e.g., images or videos) that are stored in thecontent store 210. Additionally, the social networking server 105 mayprovide application programming interface (API) functionality to senddata directly to native client device operating systems, such as IOS®,ANDROID™, or BlackberryOS.

Each user of the social networking system supported by the socialnetworking server 105 is associated with a user profile, which may bestored in the user profile store 205. A user profile includesdeclarative information about the user that was explicitly shared by theuser and may also include profile information inferred by the socialnetworking server 105. In one embodiment, a user profile includesmultiple data fields, each describing one or more attributes of thecorresponding online system user. Examples of information stored in auser profile include biographic, demographic, and other types ofdescriptive information, such as work experience, educational history,gender, hobbies or preferences, location and the like. A user profilemay also store other information provided by the user, for example,images or videos. In certain embodiments, images of users may be taggedwith information identifying the online system users displayed in animage, with information identifying the images in which a user is taggedstored in the user profile of the user. A user profile in the userprofile store 205 may also maintain references to actions by thecorresponding user performed on content items in the content store 210and stored in the action log 220.

In one embodiment, a user profile includes explicit insights andimplicit insights about a user determined by the social networkingserver 105. The social networking server 105 may determine explicitinsights about a user based on the user's activity on a social network.When a user likes, shares, follows, saves, or comments on a contentitem, the social networking server 105 gathers explicit insights aboutthe user's interests based on actions that were explicitly performed bythe user. The explicit insight represents a user's expressed interest ordislikes towards topics, objects on the social network, other users,events, applications, pages, activities, and the like. The socialnetworking server 105 may also determine implicit insights about a userbased on content read or watched by the user, demographic information,user behavior, time of day, geography, activity of the user's friends.Although not based on information directly shared by the user, implicitinsights can be inferred about the user and stored in the user profileto provide additional information about the user.

The content store 210 stores objects that each represent various typesof content. Examples of content represented by an object include a pagepost, a status update, a photograph, a video, a link, a shared contentitem, a gaming application achievement, a check-in event at a localbusiness, a brand page, or any other type of content. Online systemusers may create objects stored by the content store 210, such as statusupdates, photos tagged by users to be associated with other objects inthe social networking server 105, events, groups or applications. Insome embodiments, objects are received from third-party applications orthird-party applications separate from the social networking server 105.In one embodiment, objects in the content store 210 represent singlepieces of content, or content “items.” Hence, online system users areencouraged to communicate with each other by posting text and contentitems of various types of media to the social networking server 105through various communication channels. This increases the amount ofinteraction of users with each other and increases the frequency withwhich users interact within the social networking server 105.

In various embodiments, a content item includes various componentscapable of being identified and retrieved by the social networkingserver 105. Example components of a content item include: a title, textdata, image data, audio data, video data, a landing page, a userassociated with the content item, or any other suitable information. Thesocial networking server 105 may retrieve one or more specificcomponents of a content item for presentation in some embodiments. Forexample, the social networking server 105 may identify a title and animage from a content item and provide the title and the image forpresentation rather than the content item in its entirety.

Various content items may include an objective identifying aninteraction that a user associated with a content item desires otherusers to perform when presented with content included in the contentitem. Example objectives include: installing an application associatedwith a content item, indicating a preference for a content item, sharinga content item with other users, interacting with an object associatedwith a content item, or performing any other suitable interaction. Ascontent from a content item is presented to online system users, thesocial networking server 105 logs interactions between users presentedwith the content item or with objects associated with the content item.Additionally, the social networking server 105 receives compensationfrom a user associated with content item as online system users performinteractions with a content item that satisfy the objective included inthe content item.

Additionally, a content item may include one or more targeting criteriaspecified by the user who provided the content item to the socialnetworking server 105. Targeting criteria included in a content itemrequest specify one or more characteristics of users eligible to bepresented with the content item. For example, targeting criteria areused to identify users having user profile information, edges, oractions satisfying at least one of the targeting criteria. Hence,targeting criteria allow a user to identify users having specificcharacteristics, simplifying subsequent distribution of content todifferent users.

In one embodiment, targeting criteria may specify actions or types ofconnections between a user and another user or object of the socialnetworking server 105. Targeting criteria may also specify interactionsbetween a user and objects performed external to the social networkingserver 105, such as on a third party system (not shown). For example,targeting criteria identifies users that have taken a particular action,such as sent a message to another user, used an application, joined agroup, left a group, joined an event, generated an event description,purchased or reviewed a product or service using an online marketplace,requested information from a third party system, installed anapplication, or performed any other suitable action. Including actionsin targeting criteria allows users to further refine users eligible tobe presented with content items. As another example, targeting criteriaidentifies users having a connection to another user or object or havinga particular type of connection to another user or object.

The action logger 215 receives communications about user actionsinternal to and/or external to the social networking server 105,populating the action log 220 with information about user actions.Examples of actions include adding a connection to another user, sendinga message to another user, uploading an image, reading a message fromanother user, viewing content associated with another user, andattending an event posted by another user. In addition, a number ofactions may involve an object and one or more particular users, so theseactions are associated with the particular users as well and stored inthe action log 220.

The action log 220 may be used by the social networking server 105 totrack user actions on the social networking server 105, as well asactions on third party systems that communicate information to thesocial networking server 105. Users may interact with various objects onthe social networking server 105, and information describing theseinteractions is stored in the action log 220. Examples of interactionswith objects include: commenting on posts, sharing links, checking-in tophysical locations via a client device 110, accessing content items, andany other suitable interactions. Additional examples of interactionswith objects on the social networking server 105 that are included inthe action log 220 include: commenting on a photo album, communicatingwith a user, establishing a connection with an object, joining an event,joining a group, creating an event, authorizing an application, using anapplication, expressing a preference for an object (“liking” theobject), and engaging in a transaction. Additionally, the action log 220may record a user's interactions with advertisements on the socialnetworking server 105 as well as with other applications operating onthe social networking server 105. In some embodiments, data from theaction log 220 is used to infer interests or preferences of a user,augmenting the interests included in the user's user profile andallowing a more complete understanding of user preferences.

The action log 220 may also store user actions taken on a third partysystem, such as an external website, and communicated to the socialnetworking server 105. For example, an e-commerce website may recognizea user of an social networking server 105 through a social plug-inenabling the e-commerce website to identify the user of the socialnetworking server 105. Because users of the social networking server 105are uniquely identifiable, e-commerce websites, such as in the precedingexample, may communicate information about a user's actions outside ofthe social networking server 105 to the social networking server 105 forassociation with the user. Hence, the action log 220 may recordinformation about actions users perform on a third party system,including webpage viewing histories, advertisements that were engaged,purchases made, and other patterns from shopping and buying.Additionally, actions a user performs via an application associated witha third party system and executing on a client device 110 may becommunicated to the action logger 215 by the application for recordationand association with the user in the action log 220.

In one embodiment, the edge store 225 stores information describingconnections between users and other objects on the social networkingserver 105 as edges. Some edges may be defined by users, allowing usersto specify their relationships with other users. For example, users maygenerate edges with other users that parallel the users' real-liferelationships, such as friends, co-workers, partners, and so forth.Other edges are generated when users interact with objects in the socialnetworking server 105, such as expressing interest in a page on thesocial networking server 105, sharing a link with other users of thesocial networking server 105, and commenting on posts made by otherusers of the social networking server 105. Edges may connect two userswho are connections in a social network, or may connect a user with anobject in the system. In one embodiment, the nodes and edges form acomplex social network of connections indicating how users are relatedor connected to each other (e.g., one user accepted a friend requestfrom another user to become connections in the social network) and how auser is connected to an object due to the user interacting with theobject in some manner (e.g., “liking” a page object, joining an eventobject or a group object, etc.). Objects can also be connected to eachother based on the objects being related or having some interactionbetween them.

An edge may include various features each representing characteristicsof interactions between users, interactions between users and objects,or interactions between objects. For example, features included in anedge describe a rate of interaction between two users, how recently twousers have interacted with each other, a rate or an amount ofinformation retrieved by one user about an object, or numbers and typesof comments posted by a user about an object. The features may alsorepresent information describing a particular object or user. Forexample, a feature may represent the level of interest that a user hasin a particular topic, the rate at which the user logs into the socialnetworking server 105, or information describing demographic informationabout the user. Each feature may be associated with a source object oruser, a target object or user, and a feature value. A feature may bespecified as an expression based on values describing the source objector user, the target object or user, or interactions between the sourceobject or user and target object or user; hence, an edge may berepresented as one or more feature expressions.

The edge store 225 also stores information about edges, such as affinityscores for objects, interests, and other users. Affinity scores, or“affinities,” may be computed by the social networking server 105 overtime to approximate a user's interest in an object or in another user inthe social networking server 105 based on the actions performed by theuser. A user's affinity may be computed by the social networking server105 over time to approximate the user's interest in an object, in atopic, or in another user in the social networking server 105 based onactions performed by the user. Computation of affinity is furtherdescribed in U.S. patent application Ser. No. 12/978,265, filed on Dec.23, 2010, U.S. patent application Ser. No. 13/690,254, filed on Nov. 30,2012, U.S. patent application Ser. No. 13/689,969, filed on Nov. 30,2012, and U.S. patent application Ser. No. 13/690,088, filed on Nov. 30,2012, each of which is hereby incorporated by reference in its entirety.Multiple interactions between a user and a specific object may be storedas a single edge in the edge store 225, in one embodiment.Alternatively, each interaction between a user and a specific object isstored as a separate edge. In some embodiments, connections betweenusers may be stored in the user profile store 205, or the user profilestore 205 may access the edge store 225 to determine connections betweenusers.

FIG. 2B is a block diagram of an architecture of a communication server,in accordance with an embodiment. The communication server 125facilitates video calls among a plurality of participants associatedwith a plurality of client devices 115 over the network 110. Video callsmay also be made using a communication system 120. Responsive to a videocall request from a caller client device 115, the communication server125 initiates a video call between the caller client device 115 and oneor more receiver client device 115 specified by the caller client device115 in the video call request.

The communication server 125 includes a recommendation module 230, acall module 235, and an in-call activity module 240. In otherembodiments, the communication server 125 may include fewer, ordifferent components for various applications. Conventional componentssuch as network interfaces, security functions, load balancers, failoverservers, management and network operations consoles, and the like arenot shown so as to not obscure the details of the system architecture.The communication server 125 may use a single processor or amulti-processor system in conjunction with a storage medium such as anon-transitory computer-readable storage medium that stores instructionsexecutable by the processor(s) to carry out various functions describedherein.

The recommendation module 230 selects one or more in-call activities toinclude in a joint recommendation for a group of participants of a videocall that are relevant to each of the participants. To encourage usersto engage in in-call activities during a video call, the recommendationmodule 230 may select recommended in-call activities that arepersonalized for the group of participants in the call. After selectingthe recommended in-call activities for the group of participants, thecommunication server 125 provides the recommendations to each of theclient devices 115 participating in the video call. The communicationserver 125 transmits an indication representing the recommendations viathe network 110 to the client devices 115. Responsive to receiving therecommendations, the client devices 115 generate a user interface todisplay graphical elements representative of the recommendations thatenables initiation of the recommended in-call activity when selectedduring the video call.

In-call activities available on the video call may include one or moreof the following: watching a video, listening to music, interactive bookreading, visual effects, games, classes (e.g., educational, cooking,exercise), reading news articles, collaborative document editing, andkaraoke. Each of the different in-call activities may be performed usingan individual application that can be executed during the video call.The in-call activities provide interactive elements to all participantsof a video call on their respective client devices 115 with facilitationfrom the communication server 125. For example, when activitiesinvolving media such as “watching a video” or “listening to music” isselected, the media associated with the activity is presented to all ofthe call participants within the video call, allowing the participantsto communicate and participate in a same activity as if they were in thesame room. When “interactive book reading” is selected, a digital bookmay be presented to each of the participants, allowing the callparticipants to read together. A participant that is reading the bookmay interact with the digital book (e.g., flip the page) and theinteraction may be carried out for all of the participants in the callas if they are all reading out of one physical book together. The bookmay be displayed in a way that allows the participants to see othercallers in one window of the video call and the book in another windowof the video call. When “visual effects” is selected, the video of theparticipants may be overlaid with a visual effect such as a filter. Theclient devices 115 may communicate with the application hosting theselected in-call activity via the communication server 125 and receivethe media through the network 110.

In one embodiment, the recommendation module 230 may select arecommended in-call activity based on information from the socialnetworking server 105 including explicit insights and implicit insightsabout participants in the video call retrieved from the user profilestore, actions performed by the participants within the social networkstored by the action log, affinity of the participants towards objectsin the social network stored by the edge store 225, as well as externalfactors such as the time of day, time of year, upcoming holidays,upcoming events, significant dates (e.g., birthday, anniversary), andweather. The user information may also include past use history ofin-call activities that shows previously selected in-call activities ofthe participants.

In an embodiment, if a user chooses to opt in to the feature, therecommendation module 230 may analyze audio signals and video signalscaptured by client devices 115 to identify additional features about theuser and the user's environment using content recognition. During avideo call, the communication server 125 may identify details about theenvironment surrounding the participants (e.g., in a kitchen, in aliving room), user characteristics, and user mood using soundrecognition or image recognition algorithms. The recommendation module230 may request permission from a participant prior to using contentrecognition and enable the user to opt out of these factors being usedto generate recommendations.

Because the recommended in-call activities for a video call depends onthe participants in the call, the recommended in-call activities may bedynamically updated as new participants are added to the call. Forexample, during a first session of a video call, the recommendationmodule 230 may select a first set of recommended in-call activities fora first participant and a second participant based on its relevancy tothe two participants. During a second session of the video call, a thirdparticipant may be added to the video call, which causes therecommendation module 230 to select a second set of recommended in-callactivities for the second session to be relevant to all threeparticipants. Similarly, the recommended in-call activities may beupdated when a participant leaves the video call.

The recommended in-call activities may be dynamically updated over time.In an embodiment, the recommendation module 230 may periodically updatethe recommendation (e.g., every 10 minutes) during a video call. If atleast one recommended in-call activity is not selected by theparticipants during a threshold time period, the recommendation module230 may update the recommendation to include a different set of in-callactivities. In one embodiment, the recommendation module 230 may receivea request from one or more client devices 115A to update therecommendation.

In one embodiment, the recommendation module 230 may analyze theexplicit insights and implicit insights of the participants in the videocall and identify overlapping interests among the participants. Therecommendation module 230 may then identify in-call activities thatcorrespond to the identified overlapping interests. In anotherembodiment, the recommendation module 230 may first identify recommendedin-call activities for each of the participants in the video callindividually. Once the recommended in-call activities for theparticipants are determined, the recommendation module 230 identifiesin-call activities that are common across the plurality of participants.To determine the recommendation for the individual participants, therecommendation module 230 may use implicit insights and explicitinsights associated with the user and identify a plurality of in-callactivities that correspond to the insights.

To determine which in-call activities to recommend, the recommendationmodule 230 may generate a score for each of the in-call activitiescorresponding to common interests of the participants. For each in-callactivity that corresponds to the common interests, the recommendationmodule 230 may generate a score for each participant. The score mayrepresent a likelihood of the participant selecting the particularin-call activity responsive to being recommended the in-call activity.The recommendation module 230 may determine an aggregate score for eachin-call activity based on the individual scores of the participants. Therecommendation module 230 may rank the in-call activities based on theaggregate scores and select one or more recommended in-call activitiesbased on the ranking.

In one embodiment, the recommendation module 230 may use a machinelearning model to select in-call activities to recommend. Therecommendation module 230 may receive input data associated with a groupof participants in a video call. The input data may include explicitinsights, implicit insights, external factors (e.g., location, time ofday), and information from content recognition. The recommendationmodule 230 applies the machine learning model that identifies one ormore recommendations of in-call activities for the group ofparticipants.

In an example of a recommendation generated using various contextualinformation, a grandparent on a first client device may be on a videocall with a grandchild on a second client device, where the secondclient device is associated with a parent of the grandchild. Thecommunication server 125 may use an age recognition algorithm and othercontent recognition algorithms to dynamically determine characteristicsof participants and identify that a child is on the call.

Continuing with the example above, the recommendation module 230 may usecontent recognition to identify features to predict relevant in-callactivities. The grandchild on the video call may be wearing pajamas andmay be in a bedroom including children's toys and information may begathered by applying content recognition algorithms to the background ofthe environment. In addition, it may be night time in the time zonewhere the client device 115 that the grandchild is calling from islocated. Given factors such as relationship between the participants(e.g., grandparents and grandchild), outfit suggesting that the child ispreparing for bed, and that it is night time where the grandchild islocated, the recommendation module 230 may recommend a story time app toallow the participants to read a bed time story together. If the timezone suggests that it is in the morning instead, the recommendationmodule 230 may recommend watching a cartoon together instead of thestory time app.

In one example, a long-distance couple may be participating in a videocall in the evening of their anniversary. Given the relationship betweenthe users and the significant date, the recommendation module 230 mayrecommend watching their wedding video together. The recommendationmodule 230 may receive information about the relationship between theusers from the social networking server 105. For example, the socialnetwork profiles of the users may include relationship status and theanniversary date of the users. The couple may also post on the socialnetwork including words such as “Happy Anniversary” and tag each other,which can indicate that they are celebrating their anniversary.

In one example, each of the participants in a video call may be gettingready for a party. The recommendation module 230 may receive calendarinformation from the social networking server 105 indicating that allthe participants have accepted an invite to a party that begins withinthe next two hours. Based on the information, the recommendation module230 may recommend an app for listening to music and suggest a playlistthat includes artists that the participants like in common.

In an embodiment, the recommendation module 230 may select an in-callactivity for video call, and after selecting the in-call activity,personalize it to the group of participants in the video call. Forexample, the recommendation module 230 may first determine that thein-call activities of “reading” or “listening to music” may be relevantfor the participants. The recommendation module 230 may then customizethe activity by selecting a particular book to recommend for the“reading” recommendation or selecting a particular album for the“listening to music” recommendations. For example, if the participantsrecently checked into a concert together, the recommendation module 230may suggest listening to new album from the artist that performed at theconcert.

The recommendation module 230 may recommend in-call activities that arein different categories to give a variety of options to the users. Therecommendation module 230 may group candidate in-call activities intocategories (e.g., music-related, news-related, video-related) and selecta highest ranked activity within each of the categories forrecommending.

The call module 235 facilitates audio and video calls between aplurality of client devices 115. The call module 235 may receive arequest from a client device 115 of a caller to be connected to one ormore client devices 115 of one or more call receivers. The call module235 allows client devices 115 in the video call to communicate byfacilitating exchange of audio signals and video signals between theclient devices 115.

The in-call activity module 240 facilitates providing an in-callactivity to each of the participants of a video call. When an in-callactivity is selected during a video call, the in-call activity module250 may facilitate a connection between each of the client devices 115and an application associated with the in-call activity that is managedoutside of the social network. The in-call activity module 240 mayreceive a request from one or more client devices 115 to initiate theselected in-call activity. Responsive to receiving the request, thein-call activity module 240 communicates with the application associatedwith the selected in-call activity to provide the in-call activity toeach of the client devices 115.

FIG. 3A is an example user interface 300 of a video call, in accordancewith an embodiment. The user interface 300 includes a first window 305that displays video from a first client device associated with remoteusers 315A and 315B and a second window 325 that displays video from asecond client device associated with a local user 320. The first window305 may be larger in size than the second window 325 because the firstwindow 305 displays incoming video signals from the remote users 315Aand 315B to the local user 320. The user interface 300 may include anactivity control 330 that can expand and collapse based on userinteraction. When the activity control 330 is expanded, the activitycontrol 330 displays a control for each of the available in-callactivities that may be selected by users for initiating an in-callactivity for co-experiencing with other participants in the video call.

FIG. 3B is an example user interface of the video call including in-callactivity recommendations, in accordance with an embodiment. When a userinteracts with a graphical element associated with the activity control330 (e.g. by clicking, providing touch input), the activity control 330may expand and display a list of available in-call activities. In anembodiment, the activity control 330 may automatically display the listof in-call activities without requiring user interaction with theactivity control 330. The user interface 300 may include video callcontrols 335 including a mic button, a mute button, a camera orientationbutton, and an end call button. The video call controls 335 may includeadditional or different controls than what is shown in FIG. 3A.

When the recommendation module 230 of the communication server 125generates an updated recommendation, it transmits an indication of therecommended in-call activity to update the activity control 330 thatdisplays the recommended in-call activity to include the associatedin-call activity. The activity control 330 may overlap a portion of thefirst window 305 that displays video signal collected by a clientdevice. The activity control 330 may be displayed as a transparent imagesuch that the portion of the first window 305 that is overlaid with theactivity control 330 may be visible behind the activity control 330.

The activity control 330 may include one or more recommended in-callactivities as well as default activities that are always presented to auser in the activity control 330. The default activities may includeactivities that are presented to all users by default because it ispopular across users of the social network and activities that arefrequently accessed by a particular user. When the activity control 330is displayed to the different participants in the video call, positionsof the recommended in-call activities may be same for each of theparticipants or different for the participants. A particular in-callactivity may be displayed at the top for the remote users 315 butdisplayed at the bottom for the local user 320.

In one embodiment, the activity control 330 may include a graphicalelement for searching through available in-call activities. Although notshown in FIG. 3B, the graphical element may be a set of arrows or ascroll bar. When the client device receives an input from a user via thegraphical element such as a scrolling command to display a different setof in-call activities, the client device updates the user interface toremove an existing in-call activity from display and adds a new in-callactivity.

In one embodiment, the client device may receive a request for a previewof an in-call activity. For example, a cursor may indicate a position ofa user's movement within the activity control 330. As a user browsesthrough the activity control 330, the client device may visuallydistinguish an in-call activity that the cursor is pointing to (e.g.,change font, highlight). When the cursor lies over a particular in-callactivity for more than a threshold duration, the client device mayautomatically display a preview of the in-call activity. For example, ifthe cursor lies over the in-call activity of “watch funny dog videos,” athumbnail view of a dog video may appear in a new window within the userinterface 300.

In one embodiment, after a participant selects an activity, the activitymay automatically be initiated on each of the devices on the call. Inanother embodiment, if the remote users 315 select an in-activity, theuser interface 300 may display a message to a local user indicating thatthe remote user has selected the in-call activity and invite the localuser to join.

FIG. 4 is a flowchart illustrating a process for selecting an in-callactivity for participants of a video call, in accordance with anembodiment. In a process 400 for making a recommendation for in-callactivity to a plurality of participants of a video call, a communicationserver facilitates 410 a video call between a plurality of clientdevices over a network. The communication server may be associated witha social network that maintains user profiles including user informationof participants associated with the plurality of client devices. Thecommunication server retrieves 420 user information associated with eachparticipant of the plurality of participants associated with theplurality of client devices included in the video call. Based on theretrieved user information, the communication server determines 430common interests among the plurality of participants. The commoninterests may be determined based on explicit insights and/or implicitinsights about the plurality of participants. The explicit insights andthe implicit insights may be determined based on user activities on thesocial network, external factors such as date, time of day, location,and content recognition of audio and/or video signals.

The communication server uses the identified common interests toidentify 440 one or more in-call activities associated with theidentified common interests. From the candidate in-call activities, thecommunication server selects 450 a recommended in-call activity for theplurality of participants. Once the recommended in-call activity isselected, the communication server transmits 460 an indication of therecommended in-call activity to a client device to enable to the clientdevice to display a user interface control for initiating therecommended in-call activity. The communication server receives 470 aselection from the client device to initiate the recommended in-callactivity from the client device. Responsive to receiving the selection,the communication server initiates 480 the recommended in-call activityon each of the plurality of client devices in response to the selection.

Additional Considerations

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, and/or it may comprise a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in anon-transitory, tangible computer readable storage medium, or any typeof media suitable for storing electronic instructions, which may becoupled to a computer system bus. Furthermore, any computing systemsreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may comprise informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: facilitating a video callbetween a plurality of client devices over a network; retrieving userinformation associated with each participant of a plurality ofparticipants associated with the plurality of client devices included inthe video call; determining common interests among the plurality ofparticipants based on the user information; identifying a plurality ofin-call activities associated with the common interests; for eachin-call activity: generating, for each of the plurality of participants,a score representing a likelihood of the participant selecting thein-call activity; and generating an aggregate score for the in-callactivity by aggregating individual scores of the plurality ofparticipants; selecting a recommended in-call activity from theidentified plurality of in-call activities for the plurality ofparticipants based on the aggregate scores of the plurality of in-callactivities; and transmitting to a client device of the plurality ofclient devices, an indication of the recommended in-call activity toenable the client device to display a user interface control forinitiating the recommended in-call activity.
 2. The method of claim 1,further comprising: customizing contents of the recommended in-callactivity for the plurality of participants based on the common interestsin response to receiving a selection from the user interface controldisplayed on the client device to initiate the recommended in-callactivity from the client device.
 3. The method of claim 1, wherein thecommon interests are determined based on an overlap in user interestsand user activities among the plurality of participants.
 4. The methodof claim 1, wherein the common interests are determined based onlocation and current time at the location of at least one participant ofthe plurality of participants.
 5. The method of claim 1, wherein thecommon interests are determined based on relationships among theplurality of participants.
 6. The method of claim 1, wherein the commoninterests are determined based on content recognition from an audiosignal or a video signal from a client device during the video call. 7.The method of claim 1, wherein selecting the recommended in-callactivity from the identified plurality of in-call activities comprises:ranking the identified plurality of in-call activities; and selectingthe recommended in-call activity based on the ranking.
 8. The method ofclaim 1, wherein the recommended in-call activity is displayed alongwith a plurality of default in-call activities in the user interface. 9.The method of claim 7, wherein the plurality of default in-callactivities is different for at least one participant of the plurality ofparticipants.
 10. The method of claim 1, wherein the one or more in-callactivities include one or more of the following: watching a video,listening to music, interactive book reading, games, or karaoke.
 11. Amethod comprising: establishing a video call with a plurality of clientdevices; generating a video call user interface (UI) window to bedisplayed during the video call; during the video call, receiving arecommendation of an in-call activity selected from a plurality ofin-call activities for participants of the video call based on commoninterests among the participants, wherein the recommended in-callactivity is selected based on a ranking of aggregate scores of theplurality of in-call activities, wherein each aggregate score associatedwith an in-call activity is generated by aggregating individual scoresof the participants, and wherein each individual score represents alikelihood of the corresponding participant selecting the in-callactivity; and generating an in-call activity control in the UI window,the in-call activity control including the recommendation.
 12. Themethod of claim 11, further comprising: transmitting the selection ofthe in-call activity to a server; and facilitating the in-call activityduring the video call, wherein contents of the selected in-call activityare customized based on the common interests among the participants inresponse to receiving a selectin of an in-call activity from the in-callactivity control.
 13. The method of claim 11, wherein the in-callactivity control in the UI window dynamically updates over time.
 14. Themethod of claim 11, wherein the in-call activity control in the UIwindow changes when a new participant is added to the video call. 15.The method of claim 11, further comprising: receiving a scrollingcommand in the in-call activity control; and responsive to receiving therequest, scrolling through available in-call activities in the in-callactivity control.
 16. The method of claim 15, wherein an in-callactivity displayed in the in-call activity control is removed from thein-call activity control and a new in-call activity is displayed. 17.The method of claim 11, wherein the in-call activity control overlaps awindow displaying the video call.
 18. A non-transitory computer-readablemedium comprising computer program instructions that when executed by acomputer processor of an online system cause the processor to performsteps comprising: facilitating a video call between a plurality ofclient devices over a network; retrieving user information associatedwith each participant of a plurality of participants associated with theplurality of client devices included in the video call; determiningcommon interests among the plurality of participants based on the userinformation; identifying a plurality of in-call activities associatedwith the common interests; for each in-call activity: generating, foreach of the plurality of participants, a score representing a likelihoodof the participant selecting the in-call activity; and generating anaggregate score for the in-call activity by aggregating individualscores of the plurality of participants; selecting a recommended in-callactivity from the identified plurality of in-call activities for theplurality of participants based on the aggregate scores of the pluralityof in-call activities; and transmitting to a client device of theplurality of client devices, an indication of the recommended in-callactivity to enable the client device to display a user interface controlfor initiating the recommended in-call activity.
 19. The non-transitorycomputer readable medium of claim 18, further comprising instructionsthat when executed by the computer processor of the online system causethe processor to perform steps comprising: customizing contents of therecommended in-call activity for the plurality of participants based onthe common interests in response to receiving a selection from the userinterface control displayed on the client device to initiate therecommended in-call activity from the client device.
 20. Thenon-transitory computer-readable medium of claim 18, wherein the commoninterests are determined based on an overlap in user interests and useractivities among the plurality of participants.
 21. The non-transitorycomputer-readable medium of claim 18, wherein the common interests aredetermined based on location and current time at the location of atleast one participant of the plurality of participants.
 22. Thenon-transitory computer-readable medium of claim 18, wherein the commoninterests are determined based on relationships among the plurality ofparticipants.
 23. The non-transitory computer-readable medium of claim18, wherein the common interests are determined based on contentrecognition from an audio signal or a video signal from a client deviceduring the video call.